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Abstract 


The  Department  of  Defense  (DoD)  acquires  billions  of  dollars  of  supplies  and 
services  every  year.  In  fiscal  year  (FY)  2013,  the  DoD  obligated  over  $258  billion  for 
military-unique  weapon  systems  as  well  as  commercial  supplies  and  services.  An 
integral  part  of  the  DoD’s  contract  management  process  is  the  source  selection 
phase  when  offerors’  proposals  are  evaluated  and  the  contract  award  decision  is 
made.  A  critical  aspect  of  the  source  selection  phase  is  the  evaluation  of  contractor 
past  performance  information  as  part  of  the  overall  proposal  evaluation  process.  The 
DoD  uses  the  Past  Performance  Information  Retrieval  System  (PPIRS),  which 
consists  of  contractor  report  cards  extracted  from  the  Contractor  Performance 
Assessment  Reporting  System  (CPARS).  In  this  research  we  examine  the  value  of 
CPARS  report  card  narratives  for  service  contracts  as  they  relate  to  their  associated 
objective  scores.  Our  primary  focus  in  this  research  is  to  examine  if  the  CPARS 
report  card  written  narrative  section  provides  value  to  the  contractor  performance 
evaluation  process.  Our  data  analysis  includes  sentiment  and  statistical  analysis,  as 
well  as  interviews  with  government  agency  contracting  professionals.  Using  CPARS 
data,  narrative  analyses,  and  interviews,  we  answer  the  following  research 
questions:  (1)  To  what  degree  are  government  contracting  professionals  submitting 
to  CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual?  (2)  What  is  the  added  value  of  the  contractor 
performance  narratives  beyond  the  value  of  the  objective  scores  for  performance? 
(3)  What  is  the  statistical  relationship  between  the  sentiment  contained  in  the 
narratives  and  the  objective  scores  for  contractor  evaluations?  The  research 
revealed  that  there  are  a  variety  of  opportunities  to  improve  the  contracting  process 
specifically  related  to  the  narrative  portion  of  past  performance  assessment  reports. 

Keywords:  Services  Acquisition,  Services  Contracts,  Success  of  Services 
Contracts 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


About  the  Authors 


Dr.  Uday  Apte  is  Distinguished  Professor  of  Operations  Management  at  the 
Graduate  School  of  Business  and  Public  Policy  (GSBPP),  Naval  Postgraduate 
School  (NPS),  Monterey,  CA.  He  also  serves  as  the  associate  dean  of  Research 
and  Development  at  GSBPP.  Before  joining  NPS,  Dr.  Apte  taught  at  the  Wharton 
School,  University  of  Pennsylvania,  Philadelphia,  and  at  the  Cox  School  of  Business, 
Southern  Methodist  University,  Dallas.  He  is  experienced  in  teaching  a  range  of 
operations  management  and  management  science  courses  in  the  executive  and  full¬ 
time  MBA  programs.  Prior  to  his  career  in  academia,  Dr.  Apte  worked  for  over  10 
years  managing  operations  and  information  systems  in  the  financial  services  and 
utility  industries.  Since  then,  he  has  consulted  with  several  major  U.S.  corporations 
and  international  organizations. 

Areas  of  Dr.  Apte’s  research  interests  include  managing  service  operations, 
supply  chain  management,  technology  management,  and  globalization  of 
information-intensive  services.  He  has  completed  over  10  sponsored  research 
projects  for  the  U.S.  Department  of  Defense  and  has  published  over  60  articles,  five 
of  which  have  won  awards  from  professional  societies.  His  research  articles  have 
been  published  in  prestigious  journals  including  Management  Science,  Interfaces, 
Production  and  Operations  Management  (POM),  Journal  of  Operations  Management 
(JOM),  Decision  Sciences  Journal  (DSJ),  HE  Transactions,  and  MIS  Quarterly.  He 
has  co-authored  two  books,  Manufacturing  Automation  and  Managing  in  the 
Information  Economy.  Dr.  Apte  has  served  as  a  vice  president  of  colleges  at 
Production  and  Operations  Management  Society  (POMS),  as  a  founder  and 
president  of  the  POMS  College  of  Service  Operations,  and  as  guest  editor  of  POM 
journal.  Currently  he  serves  as  the  senior  editor  of  POM  and  as  associate  editor  of 
DSJ. 


Dr.  Apte  holds  a  PhD  in  decision  sciences  from  the  Wharton  School, 
University  of  Pennsylvania.  His  earlier  academic  background  includes  an  MBA  from 
the  Asian  Institute  of  Management,  Manila,  Philippines,  and  Bachelor  of  Technology 
(Chemical  Engineering)  from  the  Indian  Institute  of  Technology,  Bombay,  India. 

Uday  Apte,  PhD 

Professor  of  Operations  Management 

Graduate  School  of  Business  and  Public  Policy 

Naval  Postgraduate  School 

Monterey,  CA  93943-5197 

Tel:  (831)  656-3598 

E-mail:  umapte@nps.edu 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Dr.  Rene  G.  Rendon  is  a  nationally  recognized  authority  in  the  areas  of 
supply  management,  contract  management,  and  project  management.  He  is 
currently  on  the  faculty  of  the  U.S.  Naval  Postgraduate  School  (NPS),  where  he 
teaches  in  the  MBA  and  Master  of  Science  programs.  Prior  to  his  appointment  at 
NPS,  he  served  for  more  than  22  years  as  an  acquisition  and  contracting  officer  in 
the  United  States  Air  Force,  retiring  at  the  rank  of  lieutenant  colonel.  His  Air  Force 
career  included  assignments  as  a  warranted  contracting  officer  for  the  Peacekeeper 
ICBM,  Maverick  Missile,  C-20  (Gulfstream  IV),  and  the  F-22  Raptor.  He  was  also  a 
contracting  squadron  commander  for  an  Air  Force  pilot  training  base  and  the  director 
of  contracting  for  the  Air  Force’s  Space-based  Infrared  satellite  system,  and  the 
Evolved  Expendable  Launch  Vehicle  rocket  program. 

Rendon  has  taught  contract  management  courses  for  the  UCLA  Government 
Contracts  program;  he  was  also  a  senior  faculty  member  for  the  Keller  Graduate 
School  of  Management,  where  he  taught  MBA  courses  in  project  management  and 
contract  management.  He  is  a  graduate  of  the  U.S.  Air  Force  Squadron  Officer 
School,  Air  Command  and  Staff  College,  Air  War  College,  and  the  Department  of 
Defense  Systems  Management  College.  Rendon  is  Level  III  certified  in  both 
program  management  and  contracting  under  the  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA)  program.  He  is  also  a  certified  professional  contracts 
manager  (CPCM)  with  the  National  Contract  Management  Association  (NCMA),  a 
certified  purchasing  manager  (C.P.M.)  with  the  Institute  for  Supply  Management 
(ISM),  and  a  certified  project  management  professional  (PMP)  with  the  Project 
Management  Institute  (PMI).  He  has  received  the  prestigious  Fellow  Award  from 
NCMA,  and  he  was  recognized  with  the  United  States  Air  Force  Outstanding  Officer 
in  Contracting  Award.  He  has  also  received  the  NCMA  National  Education  Award 
and  the  NCMA  Outstanding  Fellow  Award.  Dr.  Rendon  is  a  member  of  the  ISM 
Certification  Committee  as  well  as  on  the  Editorial  Review  Board  for  the  ISM  Inside 
Supply  Management  magazine.  He  is  a  member  of  the  NCMA  Board  of  Advisors  as 
well  as  associate  editor  for  its  Journal  of  Contract  Management.  Dr.  Rendon’s 
publications  include  Government  Contracting  Basics  (2007),  U.S.  Military  Program 
Management:  Lessons  Learned  &  Best  Practices  (2007),  and  Contract  Management 
Organizational  Assessment  Tools  (2005).  He  has  also  published  scholarly  articles  in 
Contract  Management  magazine,  the  Journal  of  Contract  Management,  Program 
Manager  magazine,  Project  Management  Journal,  and  PM  Network  magazine.  He  is 
a  frequent  speaker  at  universities  and  professional  conferences  and  provides 
consulting  to  both  government  and  industry. 

Rene  G.  Rendon,  DBA 
Associate  Professor 

Graduate  School  of  Business  and  Public  Policy 
Naval  Postgraduate  School 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  IV  - 


Monterey,  CA  93945-5197 
Tel:  (831)  656-3464 
E-mail:  rgrendon@nps.edu 

Dr.  Michael  Dixon  is  an  assistant  professor  of  Operations  Management  at 
the  Graduate  School  of  Business  and  Public  Policy  at  the  Naval  Postgraduate 
School  (NPS).  He  is  interested  in  service  operations  management  and  specifically 
the  effect  of  operational  decisions  and  service  design  on  customer  experience.  Dr. 
Dixon  conducts  management  research  from  an  analytical  perspective  and  analyzes 
large  archival  data  sets  to  model  customer  behavior,  finding  ways  to  represent  the 
behavioral  aspects  of  service  design  in  quantitative  and  analytical  ways.  Specifically, 
he  is  interesting  in  understanding  the  role  of  experiential  sequence  on  participant 
perception.  In  past  research  he  analyzed  season  subscription  ticket  sales  for  a  large 
performing  arts  venue  and  showed  that  season  subscription  re-purchases  are 
influenced  by  event  sequences  from  previous  seasons  leading  to  managerial 
direction  for  event  planners  to  properly  schedule  performances  across  a  season  to 
maximize  experiential  effects.  He  teaches  managerial  statistics  to  MBA  students  at 
NPS  and  has  been  awarded  the  Liskin  Teaching  Award  two  times  in  the  past  two 
years  for  his  teaching  efforts. 

Dr.  Dixon  is  a  co-editor  of  the  Production  and  Operations  Management 
Society  (POMS)  Chronicle  newsletter  and  has  been  asked  to  serve  as  a  session 
chair  in  Decision  Sciences  Conference  in  2014.  He  has  publications  in  the  Journal  of 
Operations  Management  and  articles  under  review  at  the  Production  and  Operations 
Management,  Journal  of  Service  Management,  and  Journal  of  Service  Research.  He 
has  served  as  a  reviewer  for  Management  Science,  Production  and  Operations 
Management,  Journal  of  Operations  Management,  Journal  of  Services  Research, 
Service  Science,  and  the  Decision  Sciences  Journal  of  Innovative  Education. 

Michael  Dixon,  PhD 
Assistant  Professor 

Graduate  School  of  Business  and  Public  Policy 

Naval  Postgraduate  School 

Monterey,  CA  93945-5197 

Tel:  (831)  656-2187 

E-mail:  mjdixon@nps.edu 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


NPS-CM-14-190 


ACQUISITION  RESEARCH  PROGRAM 
SPONSORED  REPORT  SERIES 


Services  Acquisition  in  the  Department  of  Defense:  Analysis  of 
Operational  and  Performance  Data  to  Identify  Drivers  of  Success 

24  March  2015 

Dr.  Rene  G.  Rendon,  Associate  Professor 
Dr.  Uday  Apte,  Distinguished  Professor 
Dr.  Michael  Dixon,  Assistant  Professor 

Graduate  School  of  Business  &  Public  Policy 

Naval  Postgraduate  School 


Disclaimer:  The  views  represented  in  this  report  are  those  of  the  author  and  do  not  reflect  the  official  policy 
position  of  the  Navy,  the  Department  of  Defense,  or  the  federal  government. 


p  ^tSUNTIAI’ERSCIfNTlAM 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  VII  - 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Table  of  Contents 


Introduction . 1 

Research  Methodology . 2 

Literature  Review . 2 

Research  Design . 5 

Findings  and  Analysis . 6 

Summary,  Conclusions,  and  Recommendations . 10 

Summary . 10 

Conclusions . 10 

Recommendations . 11 

References . 13 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Introduction 


The  Government  Accountability  Office  (GAO)  reported  that  the  poor 
management  of  service  contracts  has  undermined  the  government’s  ability  to  obtain 
a  good  value  for  the  money  spent  and  has  contributed  to  the  GAO’s  decision  to 
designate  management  of  services  contracts  as  a  high-risk  area  for  the  DoD  (GAO, 
2013b).  In  fact,  as  stressed  in  a  recent  memorandum  for  acquisition  professionals  by 
the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD 
[AT&L]),  improving  the  efficiency  of  acquisition  of  products  and  services  is  of  utmost 
importance  to  the  DoD  (USD  [AT&L],  2010.  More  specifically,  in  a  later 
memorandum,  the  USD  (AT&L)  focused  on  “improving  tradecraft  in  services 
acquisition”  (USD  [AT&L],  2010,  p.  5)  by  strengthening  and  improving  the  services 
contracting  process.  An  important  part  of  the  services  acquisition  process  is  the 
evaluation  of  contractor  past  performance  information  using  the  Contractor 
Performance  Assessment  Reporting  System  (CPARS). 

The  CPARS  report  is  initiated  by  DoD  contracting  officers  during  the  contract 
closeout  phase  of  the  contract  management  process  for  documenting  contractor 
performance  information  on  the  completed  contract.  It  is  also  used  by  DoD 
contracting  officers  during  the  source  selection  phase  as  part  of  the  evaluation  of 
contractor  proposals.  The  CPARS  report  contains  contractor  performance 
information  using  objective  scores  in  five  categories:  Quality,  Schedule,  Cost 
Control,  Business  Relations,  and  Management  of  Key  Personnel.  In  addition  to 
these  five  objective  categories,  the  CPARS  reports  also  provide  a  subjective 
narrative  section  where  the  contracting  officer  provides  a  descriptive  narrative  of  the 
contractor’s  performance. 

Although  the  use  of  contractor  past  performance  information  is  an  important 
aspect  of  the  DoD  contract  management  process,  the  GAO  has  identified  many 
process  deficiencies  in  the  documentation  and  management  of  CPARS  reports. 

GAO  reports  have  shown  that  DoD  agencies  do  not  always  complete  the  required 
contractor  past  performance  reports  (GAO,  2007,  2009b,  2013a,  2014).  The  2012 
National  Defense  Authorization  Act  (NDAA)  required  the  DoD  to  “develop  a  strategy 
to  ensure  that  evaluations  in  past  performance  databases  used  for  making  source 
selection  decisions  are  complete,  timely,  and  accurate”  (GAO,  2014,  p.  4). 
Additionally,  the  2013  NDAA  required  a  “government-wide  strategy  to  ensure  that 
timely,  accurate,  and  complete  information  on  contractor  performance  is  included  in 
past  performance  databases  used  by  executive  agencies  for  making  source 
selection  decisions”  (GAO,  2014,  p.  1). 

Subsequently,  the  DoD  increased  focus  on  training  and  education  for 
contracting  officers,  which  resulted  in  an  increase  in  contractor  performance 
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assessments  being  completed  and  submitted.  In  2013,  the  GAO  noted  significant 
gains  in  CPARS  completion  rates:  56%  of  required  reports  were  completed  in  201 1 
while  74%  were  completed  in  2013.  However,  according  to  the  same  GAO  report, 
over  half  of  the  CPARS  reports  were  submitted  late.  More  importantly,  many  CPARS 
reports  contain  narratives  that  are  either  insufficiently  detailed  or  are  in  conflict  with 
their  associated  objective  scores.  Late  reports  lacking  sufficient  accurate  information 
provide  less-than-optimal  information  to  the  contracting  professionals  that  rely  on 
these  report  cards  for  source  selection  and  contract  administration  purposes  (GAO, 
2013a). 

The  purpose  of  this  research  is  to  determine  the  value  of  the  CPARS 
narratives  in  services  acquisition  by  comparing  the  relationships  between  the 
subjective  narratives  and  the  associated  objective  scores.  Our  analysis  allows  us  to 
suggest  improvements  to  the  CPARS  management  process,  thus  leading  to  greater 
and  more  effective  utilization  of  the  CPARS  reports  in  services  acquisition. 

Research  Methodology 

This  research  examines  the  value  of  CPARS  report  card  narratives  for  service 
contracts  as  they  relate  to  their  associated  objective  scores.  Our  primary  focus  in 
this  research  is  to  examine  if  the  CPARS  report  card  written  narrative  section 
provides  value  to  the  contractor  performance  evaluation  process.  Our  data  analysis 
includes  sentiment  and  statistical  analysis,  as  well  as  interviews  with  government 
agency  contracting  professionals.  Using  CPARS  data  collected  by  graduate  students 
Wilhite,  Stover,  and  Hart  (2013),  and  narrative  analyses  and  interviews  conducted 
by  graduate  students  Black,  Henley,  and  Clute  (2014),  we  answer  the  following 
research  questions: 

1 .  What  is  the  added  value  of  the  contractor  performance  narratives 
beyond  the  value  of  the  objective  scores  for  performance? 

2.  What  is  the  statistical  relationship  between  the  sentiment  contained  in 
the  narratives  and  the  objective  scores  for  contractor  evaluations? 

3.  To  what  degree  do  the  interview  findings  contradict,  support,  or 
enhance  the  findings  for  our  research  questions? 

Literature  Review 

Federal  procurement  policy  requires  that  agencies  collect  information 
regarding  a  contractor’s  performance  under  previously  awarded  contracts  for  all 
contracts  over  $100,000  and  make  that  information  available  for  use  in  future 
contract  award  decisions  (Nash,  Schooner,  O’Brien-Debakey,  &  Edwards,  2007). 

The  collection  of  contractor  performance  information  occurs  during  the  contract 
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closeout  phase  using  the  DoD  Contractor  Performance  Assessment  Reporting 
System  (CPARS;  Rendon  &  Snider,  2008). 

The  CPARS  assessment  data  reflects  the  contractor’s  performance  in  specific 
areas  including  quality,  schedule,  cost  control,  business  relations,  management  of 
key  personnel,  and  utilization  of  small  business.  The  “Quality”  rating  assesses  the 
contractor’s  qualitative  performance  and  compares  it  to  the  requirements  stated  in 
the  contract.  The  “Schedule”  rating  assesses  the  contractor’s  ability  to  meet 
schedules  outlined  in  the  contract  such  as  milestones,  task  orders,  delivery 
schedules,  and  administrative  requirements.  The  “Cost  Control”  rating  assesses  the 
contractor’s  ability  to  forecast,  manage,  and  control  the  costs  associated  with 
performing  contracted  services.  The  “Business  Relations”  rating  assesses  the 
contractor’s  ability  to  coordinate  its  business  activities  such  as  cooperate  behavior, 
customer  satisfaction,  management,  and  attitude  towards  customers.  The 
“Management  of  Key  Personnel”  rating  assesses  the  contactors  ability  to  maintain 
qualified  individuals  in  key  positions  as  outlined  in  the  contract.  The  “Utilization  of 
Small  Business”  rating  assesses  the  contractor’s  ability  to  integrate  small 
businesses  in  the  execution  of  the  contract  (Wilhite  et  al.,  2013). 

The  CPARS  assessment  rates  the  contractor  in  these  areas  using  the  rating 
scales  Exceptional,  Very  Good,  Satisfactory,  Marginal,  and  Unsatisfactory.  It  should 
be  noted  that  the  contractor  is  allowed  to  review  the  CPARS  assessment  and 
provide  comments  back  to  the  government  assessing  official  prior  to  the  government 
finalizing  the  CPARS  report. 

During  the  source  selection  phase  of  government  negotiated  procurement, 
contractor  performance  information  is  used  in  evaluating  offerors  and  in  making  a 
contract  award  decision  (Rendon  &  Snider,  2008).  In  this  phase,  the  government 
agency  accesses  the  contractor  performance  information  through  the  DoD  Past 
Performance  Information  Retrieval  System  Report  Cards  (PPIRS-RC)  database. 
During  source  selection  in  the  evaluation  of  offeror’s  proposals,  the  government 
agency  uses  the  contractor  past  performance  information  to  determine  if  the  offeror 
meets  the  required  standards  of  responsibility  as  stated  in  the  federal  procurement 
policy,  and,  depending  on  the  basis  of  award  stipulated  in  the  solicitation,  uses  the 
contractor’s  past  performance  ratings  to  justify  an  award  to  a  higher-priced  offeror. 

The  contractor  performance  information  reported  in  CPARS  and  accessible 
through  PPIRS  provides  outcome-based  data  that  can  be  used  to  identify  successful 
contracts.  The  successful  contracts  determined  by  using  contractor  performance 
information  have  been  used  in  our  previous  research  to  identify  the  contract 
variables  that  lead  to  contract  success. 

In  2014,  with  the  assistance  of  our  graduate  students  (Wilhite,  Hart,  &  Stover, 
2013),  we  accessed  the  past  performance  database  to  collect  contractor 
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performance  ratings  on  715  completed  Army  services  contracts  to  determine  if  the 
contracts  were  successful  or  not  successful.  Using  statistical  analysis  we 
investigated  whether  certain  contracting  variables  such  as  type  of  service,  contract 
dollar  value,  level  of  competition,  and  contract  type  affected  the  success  of  the 
contract.  The  detailed  results  of  our  analysis  are  presented  in  Rendon,  Apte,  & 

Dixon  (2014),  while  the  salient  findings  are  summarized  as  follows: 

•  The  S  type  services  (Utilities  and  Housekeeping)  had  the  highest 
failure  rate  of  all  the  product  service  codes  analyzed.  The  S  type 
services  had  1 1  contract  failures  resulting  in  a  3.77%  failure  rate.  The 
reasons  for  contract  failure  included  business  relations  and 
management  of  key  personnel  (Wilhite  et  al.,  2013). 

•  Contracts  with  a  dollar  value  from  $50  million  to  $1  billion  had  the 
highest  failure  rate  of  all  the  contract  categories.  Contracts  with  dollar 
values  between  $50  million  to  $1  billion  category  consisted  of  92 
contracts  with  eight  labeled  failures,  giving  it  a  failure  rate  of  8.7%.  This 
group’s  most  common  reason  for  failing  was  cost  control  (Wilhite  et  al., 
2013). 

•  With  respect  to  competition  in  contracting,  we  found  that  contracts 
competed  competitively  had  the  highest  failure  rate  when  compared  to 
the  other  two  forms  of  competition  available.  Of  the  540  competitive 
contracts,  17  were  labeled  as  failures,  which  yields  a  failure  rate  of 
3.15%.  The  reasons  that  most  often  resulted  in  a  contract  failure  were 
in  the  areas  of  schedule  and  cost  control  (Wilhite  et  al.,  2013). 

•  Concerning  type  of  contract,  we  found  that  contracts  structured  as  a 
combination  contract  had  the  highest  failure  rate  when  compared  to 
the  other  five  types  of  available  contracts.  There  were  four 
Combination  contracts  examined  in  the  database.  Of  these  four 
contracts,  two  were  labeled  failures,  which  yields  a  failure  rate  of 
50.0%.  Schedule  and  cost  were  both  referenced  twice  in  the  failed 
contracts  while  quality  and  management  of  key  personnel  were  each 
referenced  once  (Wilhite  et  al.,  2013).  Furthermore,  the  results  of  our 
significance  testing  showed  that  Contractual  Amounts  and  Contract 
Type  were  our  only  statistically  significant  variables  (Wilhite  et  al., 
2013). 

We  also  looked  at  the  relationships  between  percent  of  filled  contract 
specialist  (1 102)  billets  and  failure  rates,  and  between  workload  dollars  per  filled 
billet  and  failure  rates,  and  made  some  interesting  observations.  We  found  that  as 
the  percentage  of  1 102  filled  billets  increased,  the  contract  failure  rate  decreased 
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(Wilhite  et  al.,  2013).  This  would  seem  intuitive:  that  as  the  workforce  increases,  the 
contract  success  rate  would  also  increase,  since  there  would  be  sufficient  resources 
to  manage  the  contracting  process.  However,  we  also  observed  that  as  workload 
dollars  per  filled  billet  increased,  contractor  performance  ratings  also  increased,  and 
thus  contract  failure  ratings  decreased  (Wilhite  etal.,  2013).  This  relationship 
seemed  counterintuitive,  since  additional  workload  in  the  organization  would  typically 
place  a  higher  demand  on  the  workforce,  thus  resulting  in  fewer  resources  to 
manage  the  contracting  process.  One  plausible  hypothesis  explaining  this 
counterintuitive  result  is  that  “workload  dollars  per  filled  billet”  are  not  good  surrogate 
measures  of  the  actual  “average  workload  per  billet.”  We  hope  to  investigate  this 
hypothesis  further  in  our  future  research. 

Our  past  research  using  CPARS  data  identified  some  interesting  areas 
worthy  of  further  exploration.  These  areas  include  analyzing  the  narrative  portion  of 
the  CPARS  ratings  to  determine  alignment  with  the  objective  ratings,  as  well  as  the 
value  added,  not  only  in  the  narrative  portions,  but  also  in  the  usefulness  of  the 
CPARS  as  a  contractor  assessment  tool.  This  is  the  focus  of  our  current  research 
project. 

Research  Design 

Our  research  examines  the  value  of  the  CPARS  report  card  narratives  for 
service  contracts  as  they  relate  to  their  associated  objective  scores.  The  primary 
focus  in  this  research  is  examining  if  the  CPARS  report  card  written  narrative  section 
provides  value  to  the  contractor  performance  evaluation  process.  Our  data  analysis 
included  a  sentiment  analysis  and  statistical  analysis,  as  well  as  interviews  with 
government  agency  contracting  professionals. 

With  the  assistance  of  our  most  recent  MBA  thesis  students  (Black,  Henley, 
and  Clute),  we  performed  a  sentiment  analysis  of  the  715  Army  service  contract 
CPARS  report  card  narratives  accessed  in  our  previous  research  (Rendon  et  al., 
2014).  Our  students  used  the  CPARS  Ouality  Checklist  as  a  basis  for  developing  the 
criteria  for  the  categories  and  values  for  the  sentiment  analysis  (CPARS  Best 
Practices,  Ouality  Checklist,  n.d.).  In  the  sentiment  analysis,  the  student  researchers 
scored  each  narrative  along  the  dimensions  of  quality,  robustness,  compliance  with 
directions  in  the  CPARS  Ouality  Checklist,  and  its  value  and  content  compared  to  its 
related  objective  scores  from  the  CPARS  report  cards.  Independent  researchers’ 
scores  were  compared  across  a  small  sample  to  ensure  inter-rater  reliability. 

We  conducted  a  statistical  analysis  of  the  relationship  between  the  sentiment 
analysis  scores  and  their  associated  objective  rating  scores.  This  analysis 
investigated  correlating  relationships  between  the  sentiment  scores  and  the 
objective  rating  scores  for  the  same  CPARS  report.  Our  purpose  was  to  explore  the 
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relationships  between  the  sentiment  scores  and  the  objective  rating  scores  to  reveal 
the  extent  of  the  value  of  the  narratives. 

Our  students  (Black  et  al.,  2014)  also  conducted  interviews  with  contracting 
professionals  from  two  DoD  contracting  agencies.  These  interviews  focused  on  the 
agencies’  use  of  CPARS  and  other  sources  of  contractor  past  performance 
information  and  agencies’  value  of  the  CPARS  narratives  compared  to  the  objective 
rating  scores. 

Findings  and  Analysis 

In  this  section,  we  present  an  analysis  of  our  findings.  The  primary  purpose  of 
this  research  was  to  determine  the  value  of  the  CPARS  narratives  in  services 
acquisition  by  comparing  the  relationships  between  the  subjective  narratives  and  the 
objective  scores.  We  first  present  the  findings  of  the  sentiment  and  statistical 
analysis  by  focusing  on  each  of  the  criteria  used  in  the  analysis. 

1.  Do  the  narratives  address  all  performance  areas  assessed?  Overall, 
the  narratives  address  all  performance  areas  assessed  -82%  of  the  time.  This  is 
less  problematic  with  unsuccessful  contracts  at  -95%  than  with  successful  contracts 
at  -81%.  The  difference  in  the  proportion  of  times  that  the  narrative  addresses  all 
performance  areas  assessed  in  successful  and  unsuccessful  contracts  is  statistically 
significant  (p  <  .05;  Black  et  al.,  2014,  p.  41). 

2.  Are  narratives  based  on  objective  data?  Overall,  the  narratives  are 
based  on  objective  data  -77%  of  the  time.  However,  in  unsuccessful  contracts,  the 
narratives  are  based  on  objective  data  100%  of  the  time.  This  is  significantly 
different  from  the  -77%  in  successful  contracts  (p  <  .01 ;  Black  et  al.,  2014,  p.  41 ). 

3.  Are  narratives  free  of  statements  to  avoid?  Overall,  the  narratives  are 
free  of  statements  to  avoid  -97%  of  the  time.  This  is  slightly  more  problematic  with 
unsuccessful  contracts  at  -86%  than  with  successful  contracts  at  -97%  (p  <  .01 ; 
Black  et  al.,  2014,  p.  41). 

4.  Are  narratives  robust  and  comprehensive?  Overall,  the  narratives  are 
robust  and  comprehensive  -63%  of  the  time.  This  is  less  problematic  with 
unsuccessful  contracts  at  -91%  than  with  successful  contracts  at  -62%  (p  <  .01; 
Black  et  al.,  2014,  p.  41). 

5.  Could  a  layman  understand  the  description  of  the  work  performed? 

Overall,  the  narratives  are  written  so  that  a  contracting  layman  should  understand 
the  work  performed  -64%  of  the  time.  This  is  less  problematic  with  unsuccessful 
contracts  at  -82%  than  it  is  with  successful  contracts  at  -64%  (p  <  .05;  Black  et  al., 
2014,  p.  41). 
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6.  Is  the  narrative  beneficial  above  and  beyond  objective  scores?  Using 
a  Chi  Square  Test,  we  determined  that  there  was  a  difference  between  successful 
and  unsuccessful  contracts  in  whether  the  narratives  were  beneficial  above  and 
beyond  the  objective  scores.  Unsuccessful  contracts  tended  to  have  more  beneficial 
CPARS  report  card  narratives  than  successful  contracts  (Black  et  al.,  p.  42).  Overall, 
the  narrative  provides  an  unsatisfactory  amount  of  beneficial  data  to  the  user  -12% 
of  the  time.  However,  there  were  no  unsuccessful  contracts  that  provided  an 
unsatisfactory  amount  of  beneficial  data.  The  narrative  provides  a  marginal  amount 
of  beneficial  data  -22%  of  the  time.  There  were  no  unsuccessful  contracts  that 
provided  a  marginal  amount  of  beneficial  data.  The  narrative  provides  a  satisfactory 
amount  of  beneficial  data  -28%  of  the  time.  The  narrative  provides  a  very  good 
amount  of  beneficial  data  -21%  of  the  time.  The  narrative  provides  an  exceptional 
amount  of  beneficial  data  -18%  of  the  time.  This  is  much  more  likely  to  occur  with 
unsuccessful  contracts  than  with  successful  contracts  at  -17%  (Black  et  al.,  2014,  p. 
42). 

7.  Do  the  narratives  correlate  to  the  objective  scores  assigned?  Using 
the  Chi  Square  Test,  we  determined  that  there  was  not  a  difference  between 
successful  and  unsuccessful  contracts  in  whether  the  narrative  correlates  to  the 
objective  scores  assigned.  Overall,  the  narrative  sentiment  is  contradictory  to  more 
than  one  of  the  objective  scores  assigned  -2%  of  the  time.  The  narrative  sentiment 
is  contradictory  to  one  of  the  objective  scores  assigned  -6%  of  the  time.  The 
narrative  sentiment  is  satisfactory  in  describing  accurately  why  the  objective  scores 
are  assigned  as  they  are  -28%  of  the  time.  The  narrative  sentiment  is  very 
successful  in  describing  accurately  why  the  objective  scores  are  assigned  as  they 
are  -40%  of  the  time.  The  narrative  sentiment  is  exceptionally  successful  in 
describing  accurately  why  the  objective  scores  are  assigned  as  they  are  -24%  of  the 
time  (Black  et  al.,  2014,  p.  42). 
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Figure  1  summarizes  the  results  of  the  statistical  analysis. 


Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful  vs 

Unsuccessful 

Statistically 

Significant? 

P  Value 

Narrative 

Addresses  All 

Performance 

Areas  Assessed? 

No 

18.46% 

4.55% 

18.90% 

Yes 

0.043 

Yes 

81.54% 

95.45% 

81.10% 

Narrative  Based 

On  Objective 
Data? 

No 

22.52% 

0.00% 

23.23% 

Yes 

<.01 

Yes 

77.48% 

100.00% 

76.77% 

Narrative  Is  Free 

Of  Statements  To 

Avoid? 

No 

3.08% 

13.64% 

2.74% 

Yes 

<.01 

Yes 

96.92% 

86.36% 

97.26% 

Narrative  Is 

Robust  & 
Comprehensive? 

No 

37.48% 

9.09% 

38.38% 

Yes 

<.01 

Yes 

62.52% 

90.91% 

61.62% 

Could  A  Layman 
Understand  Work 

Performed? 

No 

35.52% 

18.18% 

36.08% 

Yes 

0.042 

Yes 

64.48% 

81.82% 

63.92% 

Is  The  Narrative 

Beneficial  Above 

&  Beyond 
Objective  Scores? 

Score  1 

11.61% 

0.00% 

12.12% 

Yes 

<.01 

Score  2 

21.68% 

0.00% 

22.37% 

Score  3 

27.97% 

18.18% 

28.28% 

Score  4 

20.84% 

27.27% 

20.63% 

Score  5 

17.76% 

54.55% 

16.59% 

Does  The 

Narrative 

Correlate  To 

Objective  Scores 
Assigned? 

Score  1 

1.82% 

4.55% 

1.88% 

No 

0.141 

Score  2 

6.43% 

4.55% 

6.49% 

Score  3 

27.83% 

9.09% 

28.43% 

Score  4 

40.42% 

40.91% 

40.40% 

Score  5 

23.36% 

40.91% 

22.80% 

Figure  1 .  Results  of  Statistical  Database  Analysis 

(Black  etal.,  2014) 

As  previously  discussed,  our  students  also  conducted  interviews  with 
contracting  professionals  from  two  DoD  contracting  agencies  (Black  et  al.,  2014) 
These  interviews  focused  on  the  agencies’  use  of  CPARS  and  other  sources  of 
contractor  past  performance  information  as  well  as  these  agencies’  value  of  the 
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CPARS  narratives  compared  to  the  objective  rating  scores  in  the  source  selection 
process.  The  findings  of  these  interviews  are  summarized  as  follows: 

1 .  CPARS  is  still  often  not  reliable,  robust,  or  comprehensive  enough. 

This  results  in  source  selection  officials  not  placing  a  significant 
amount  of  weight  on  the  past  performance  evaluation  criteria  (Black  et 
al„  2014,  p.  44). 

2.  Unsuccessful  contracts  tend  to  have  more  reliable,  robust,  and 
comprehensive  past  performance  information  available  in  their 
CPARS/PPIRS  reports  (Black  et  al.,  2014,  p.  45). 

3.  The  appropriate  amount  of  weight  that  should  be  assigned  to  the  past 
performance  evaluation  criteria  in  making  a  source  selection  decision 
should  be  correlated  to  the  source,  availability,  quality,  and  relevancy 
of  the  past  performance  information  (Black  et  al.,  2014,  p.  45). 

4.  The  information  found  in  PPIRS  sometimes  contains  information  in  the 
narrative  that  is  either  contradictory  or  does  not  quite  match  up  with  the 
objective  scores.  When  the  objective  scores  and  narrative  sentiment  in 
PPIRS  is  mismatched,  contracting  professionals  tend  to  give  more 
weight  to  the  narrative  versus  the  objective  scores  (Black  et  al.,  2014, 
p.  46). 

5.  Contracting  professionals  are  not  always  applying  due  diligence  in 
identifying  the  appropriate  contractor  entity  (e.g.,  CAGE  Code  or 
DUNS  number)  in  the  CPARS  reports.  This  is  resulting  in  contractor 
past  performance  information  not  being  fully  accessible  in  PPIRS 
(Black  et  al.,  2014,  p.  46). 

6.  There  is  a  lack  of  reliable,  robust,  and  comprehensive  amount  of  past 
performance  information  available  in  PPIRS.  This  results  in  source 
selection  officials  soliciting  contractors  for  references  or  asking 
contractors  to  fill  out  a  past  performance  questionnaire  (Black  et  al., 
2014,  p.  47). 

7.  The  results  of  the  interviews  also  identified  recommendations  for 
improving  the  quality  of  CPARS  reports,  incorporating  data  analytics 
tools  into  the  PPIRS  database,  enhancing  the  monitoring  of 
Contracting  Officer  Representative  (COR)  workload,  improving 
acquisition  workforce  training  on  developing  CPARS  narratives,  and 
improving  the  disclosure  of  CPARS  program  office  Audit  results  (Black 
et  al.,  2014,  pp.  48-49). 
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Summary,  Conclusions,  and  Recommendations 

Summary 

The  Department  of  Defense  (DoD)  acquires  billions  of  dollars  of  supplies  and 
services  every  year.  In  FY  2013,  the  DoD  obligated  over  $258  billion  for  military- 
unique  weapon  systems  as  well  as  commercial  supplies  and  services  (USA 
Spending,  2013).  An  integral  part  of  the  DoD’s  contract  management  process  is  the 
source  selection  phase  when  offerors’  proposals  are  evaluated  and  the  contract 
award  decision  is  made.  A  critical  aspect  of  the  source  selection  phase  is  the 
evaluation  of  contractor  past  performance  information  as  part  of  the  overall  proposal 
evaluation  process.  The  DoD  uses  the  Past  Performance  Information  Retrieval 
Systems  (PPIRS)  which  consists  of  contractor  report  cards  extracted  from  the 
Contractor  Performance  Assessment  Reporting  System  (CPARS).  Although  the  use 
of  contractor  past  performance  information  is  an  important  aspect  of  the  DoD 
contract  management  process,  the  Government  Accountability  Office  (GAO)  has 
identified  many  process  deficiencies  in  the  documentation  and  management  of 
CPARS  reports.  GAO  reports  have  shown  that  DoD  agencies  do  not  always 
complete  the  required  contractor  past  performance  reports  (GAO,  2007,  2009b, 
2013a,  2014).  More  importantly,  many  CPARS  reports  contain  narratives  that  are 
either  insufficiently  detailed  or  conflict  with  their  associated  objective  scores.  Late 
reports  lacking  sufficient  accurate  information  provide  less-than-optimal  information 
to  the  contracting  professionals  that  rely  on  these  report  cards  for  source  selection 
decisions. 

The  purpose  of  this  research  was  to  determine  the  value  of  the  CPARS 
narratives  in  services  acquisition  by  comparing  the  relationships  between  the 
subjective  narratives  and  the  objective  scores.  Our  primary  focus  in  this  research 
was  examining  if  the  CPARS  report  card  written  narrative  section  provides  value  to 
the  contractor  performance  evaluation  process.  Our  data  analysis  included 
sentiment  and  statistical  analysis,  as  well  as  interviews  with  government  agency 
contracting  professionals. 

Conclusions 

Using  CPARS  data  collected  by  graduate  students  Hart,  Stover,  and  Wilhite, 
(2013),  and  narrative  analyses  and  interviews  conducted  by  graduate  students 
Black,  Henley,  and  Clute  (2014),  we  answered  the  following  research  questions: 

1.  What  is  the  added  value  of  the  contractor  performance  narratives 
beyond  the  value  of  the  objective  scores  for  performance?  Contracting 
professionals  are  doing  a  better  job  at  providing  beneficial  CPARS  data  in  the 
narrative  when  the  contract  is  unsuccessful  versus  when  it  is  successful.  Only  38.6% 
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of  the  observed  CPARS  narratives,  whether  successful  or  unsuccessful,  provided  a 
very  good  or  exceptional  amount  of  beneficial  data  above  and  beyond  what  could  be 
gleaned  from  looking  over  the  objective  scores  assigned  (Black  et  al. ,  2014,  p.  51 ). 

2.  What  is  the  statistical  relationship  between  the  sentiment  contained 
in  the  narratives  and  the  objective  scores  for  contractor  evaluations? 

Contracting  professionals  are  developing  CPARS  narratives  that  contradict  at  least 
one  of  the  objective  scores  assigned  -8.3%  of  the  time.  Contracting  professionals 
were  slightly  better  at  matching  the  narrative  sentiment  to  the  objective  scores  in 
unsuccessful  contracts  (-81 .8%  scoring  either  very  good  or  exceptional)  than  in 
successful  contracts  (-63.2%  scoring  either  very  good  or  exceptional;  Black  et  al., 
2014,  p.  51). 

3.  To  what  degree  do  the  interview  findings  contradict,  support,  or 
enhance  the  findings  for  our  research  questions?  The  results  of  the  interviews 
found  that  the  CPARS  database  is  still  often  not  reliable,  robust,  or  comprehensive 
enough.  We  also  found  that  unsuccessful  contracts  tend  to  have  more  reliable, 
robust,  and  comprehensive  past  performance  information  available  in  their 
CPARS/PPIRS  reports.  Additionally,  the  appropriate  amount  of  weight  that  should 
be  assigned  to  the  past  performance  evaluation  criteria  in  making  a  source  selection 
decision  should  be  correlated  to  the  source,  availability,  quality,  and  relevancy  of  the 
past  performance  information.  Our  interviewees  also  stated  that  the  information 
found  in  the  PPIRS  database  sometimes  contains  information  in  the  narrative  that  is 
either  contradictory  or  does  not  quite  match  up  with  the  objective  scores.  We  also 
found  that  contracting  professionals  are  not  always  applying  due  diligence  in 
identifying  the  appropriate  contractor  entity  in  the  CPARS  reports,  which  is  resulting 
in  a  lack  of  reliable,  robust,  and  comprehensive  amount  of  past  performance 
information  available  in  PPIRS.  Finally,  the  interview  results  also  identified 
recommendations  for  improving  the  quality  of  CPARS  reports,  incorporating  data 
analytics  tools  into  the  PPIRS  database,  enhancing  the  monitoring  of  COR  workload, 
improving  acquisition  workforce  training  on  developing  CPARS  narratives,  and 
improving  the  disclosure  of  CPARS  program  office  audit  results  (Black  et  al.,  2014, 
pp.  44-49). 

Recommendations 

Based  on  our  conclusions,  we  identified  the  following  five  recommendations: 

Recommendation  1:  Training.  Training  should  be  implemented  for  all 
services  acquisition  members  that  interact  with  the  CPARS  and  PPIRS  databases. 
Training  should  focus  on  developing  comprehensive  narratives  ensuring  that 
acquisition  team  members  can  fully  understand  the  work  performed,  address  all 
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performance  areas  assessed  in  their  objective  scores,  and  ensure  the  narratives  are 
based  on  objective  data  (Black  et  al.,  2014,  pp.  54-55). 

Recommendation  2:  Process  Improvement.  The  DoD  needs  to  improve  the 
quality  of  past  performance  report  submissions  in  CPARS  and  PPIRS,  improving  the 
source,  availability,  quality,  relevancy,  and  accuracy  of  the  past  performance 
information.  This  will  allow  acquisition  teams  to  assign  higher  weights  to  past 
performance  evaluation  criteria  in  source  selection  decisions  (Black  et  al.,  2014,  pp. 
54-55). 

Recommendation  3:  Data  Analytics.  Additional  data  analysis  tools  should 
be  incorporated  into  the  CPARS  and  PPIRS  database  to  better  assist  contracting 
professionals  in  identifying  past  performance  trends  for  a  particular  contractor  or 
specific  type  of  service  (Black  et  al.,  2014,  pp.  54-55). 

Recommendation  4:  Customer  Feedback.  The  CPARS  process  should 
include  customer  feedback  on  contractor  performance.  Currently,  only  the 
acquisition  team  provides  input  to  the  CPARS  report  card.  Customer  input  into 
CPARS  will  encourage  the  submission  of  more  accurate  and  robust  CPARS  report 
cards  (Black  et  al.,  2014,  pp.  54-55). 

Recommendation  5:  COR  Manning  levels.  Contracting  Officer 
Representative  (COR)  manning  levels  should  be  reviewed  throughout  the  DoD  to 
ensure  that  organizations  have  sufficiently  filled  COR  billets  to  manage  the  CPARS 
process  (Black  et  al.,  2014,  pp.  54-55). 
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